{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "import_and_maybe_install <- function(pkgs){\n",
    "    missing_packages <- character(0)\n",
    "    for(pkg in pkgs){\n",
    "        tryCatch({\n",
    "            require(pkg, character.only=TRUE )\n",
    "        },warning=function(err){\n",
    "            missing_packages <<- c(missing_packages,pkg)\n",
    "        })\n",
    "    }\n",
    "    if(length(missing_packages)){\n",
    "            install.packages(missing_packages)\n",
    "            sapply(missing_packages, require, character.only=TRUE)\n",
    "    }\n",
    "    invisible(NULL)\n",
    "}\n",
    "import_and_maybe_install(c(\"data.table\", \"feather\", \"ggplot2\",\"clusterSEs\",\"formula.tools\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR <- \"~/connector/20200226/analytic_datasets\"\n",
    "\n",
    "commute_ride_data <- read_feather( file.path(DATA_DIR,\"commute_ride_data.feather\"))\n",
    "setDT(commute_ride_data)\n",
    "\n",
    "# FACTOR VARIABLES\n",
    "stopifnot(commute_ride_data[,class(fAid) == \"factor\"]) # rider id\n",
    "stopifnot(commute_ride_data[,class(fDate) == \"factor\"]) # date id\n",
    "\n",
    "route_field <- sprintf(\"route_%s_10_ride\",1:21)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "3354470"
      ],
      "text/latex": [
       "3354470"
      ],
      "text/markdown": [
       "3354470"
      ],
      "text/plain": [
       "[1] 3354470"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "9926"
      ],
      "text/latex": [
       "9926"
      ],
      "text/markdown": [
       "9926"
      ],
      "text/plain": [
       "[1] 9926"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "1793450"
      ],
      "text/latex": [
       "1793450"
      ],
      "text/markdown": [
       "1793450"
      ],
      "text/plain": [
       "[1] 1793450"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "7996"
      ],
      "text/latex": [
       "7996"
      ],
      "text/markdown": [
       "7996"
      ],
      "text/plain": [
       "[1] 7996"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "commute_ride_data[,.N]\n",
    "commute_ride_data[,1,aid][,.N]\n",
    "\n",
    "commute_ride_data[(!SR520WBHOV) &  (!SR520EBHOV),.N]\n",
    "commute_ride_data[(!SR520WBHOV) &  (!SR520EBHOV),1,aid][,.N]\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Dictionary\n",
    "\n",
    "Each record represents one rider commute opportunty (rider / date / commute).\n",
    "\n",
    "**Note that** there is one record for each commute / data for every rider \n",
    "*After their first ride date*.  This is because the choice to ride (`did_ride`) \n",
    "is conditioned on the rider's history of riding, and which is not available \n",
    "on thier first ride.\n",
    "\n",
    "#### Identifiers\n",
    "* **aid**: Rider number (anonomyzed Numeric)\n",
    "* **fAid**: Rider number (anonomyzed Factor)\n",
    "* **date**: Date of the ride (as Date)\n",
    "* **fDate**: Date of the ride (as Factor)\n",
    "* **AfternoonId**: Function of Date, only true for Route 5\n",
    "\n",
    "#### Main outcome / predictor\n",
    "* **did_ride**: Did the rider ride on this date?\n",
    "* **imputed_new_buses**: The weighted average of the fraction of new busses \n",
    "     on the routes the rider took on the last (up to) 10 trips\n",
    "\n",
    "#### Time since last ride\n",
    "* **commutes_since_last_ride**: Number of commutes since rider's last ride (numeric; range 1 - 461)\n",
    "* **f_commutes_since_last_ride**: Number of commutes since rider's last ride (factor; range \"1\" - \"40+\")\n",
    "     \n",
    "#### Date based indicators\n",
    "* **SR520WBHOV**: SR 520 West Bound is open (Function of the date)\n",
    "* **SR520EBHOV**: SR 520 East Bound is open (Function of the date)\n",
    "* **I405HOV**: Is a 405 HOV is open (Function of the date )\n",
    "* **Route5Expansion**: Route 5 expanded (Function of Date )\n",
    "\n",
    "#### Features based on the rider's history\n",
    "* **route_X_10_ride** (e.g. `route_3_10_ride`):  Fraction of the rider's last 10 rides that were taken on route X\n",
    "* **is_peak_10_ride**: Fraction of the rider's last 10 rides that were taken **during peak traffic**\n",
    "* **IsESOther_10_ride**: Fraction of the rider's last 10 rides that were taken on **EastSide (other) routes**\n",
    "* **IsRouteSeattle_10_ride**: Fraction of the rider's last 10 rides that were taken on **Seattle Routes**\n",
    "* **Is405North_10_ride**: Fraction of the rider's last 10 rides that were taken on **I 405 North Routes**\n",
    "* **IsCoach_10_ride**: Fraction of the rider's last 10 rides that were taken on  **Coach (as opposed to shuttle) busses**\n",
    "* **first_stop_10_ride**: Fraction of the rider's last 10 rides that were taken from **the last AM stop**\n",
    "* **last_stop_10_ride**: Fraction of the rider's last 10 rides that were taken on  **the first AM stop**\n",
    "* **single_stop_10_ride**: Fraction of the rider's last 10 rides that were taken on **a Single Stop Route**\n",
    "* **vehicle_is_new_10_ride**: Fraction of the rider's last 10 rides that were taken on **a new vehicle**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Handy Utility function for fitting models and gathering summary statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "#' A cute little function for fitting a model and pulling out the headline statistics\n",
    "#'\n",
    "#' @param data A dataset for fitting the model\n",
    "#' @param model A number for identifying the model in a table (these second field in the \"coef\" output)\n",
    "#' @param lhs The left hand side of the model (as a string)\n",
    "#' @param predictors The effects of interested.  Specifically these are the effects that will be pulled out in the summary table\n",
    "#' @param adjust_for The fixed effects that will be included in the right hand side but not pulled out into the summar table\n",
    "#'\n",
    "#' @return A list containing the glm model (`$model`), and a matrix of interesting statistics from the\n",
    "#'       various models (`$coef`)\n",
    "\n",
    "fit_ridership_logit_models <- function(data,\n",
    "                                        model=1,\n",
    "                                        lhs=\"did_ride\",\n",
    "                                        predictors=\"new_fraction_10day\",\n",
    "                                        adjust_for=c(\"fDate\",route_field),# up to 231 levels\n",
    "                                        quick=FALSE,\n",
    "                                        granularity=\"\"){\n",
    "\n",
    "    \n",
    "    # build and test the formula \n",
    "    (FMLA <- formula(paste(lhs,'~',paste(c(predictors,adjust_for),collapse=' + '))))\n",
    "    .updated = as.character(update(FMLA,~.))\n",
    "    if(length(.updated)>1){\n",
    "        .rhs = .updated[3]\n",
    "    }else{\n",
    "        .rhs = strsplit(as.character(update(FMLA,~.)),\" *~ *\")[[1]][2]\n",
    "    }\n",
    "    .terms = strsplit(.rhs,\" *\\\\+ *\")[[1]]\n",
    "    \n",
    "    \n",
    "    .vars = get.vars(rhs(FMLA))\n",
    "    if(any(! (.vars %in% names(data))))\n",
    "           stop(paste(\"missing variable: \",.vars[! (.vars %in% names(data))],collapse=\"\\\\n\"))\n",
    "           \n",
    "    if(any(!(predictors %in% .terms))){\n",
    "        cat(\"\\n===============================================\\n\")\n",
    "        cat(\"terms :\\n\")\n",
    "        cat(paste0(.terms,\"\\n\"))\n",
    "        cat(\"\\npredictors :\\n\")\n",
    "        cat(paste0(predictors[!(predictors %in% .terms)],\"\\n\"))\n",
    "        cat(\"\\n===============================================\\n\")\n",
    "        stop(\"Bad Predctors\")\n",
    "        \n",
    "    }    \n",
    "    \n",
    "    quick_N <- 5e5\n",
    "    if(quick & data[,.N] > quick_N)\n",
    "        data <- data[sample(.N,quick_N)]\n",
    "    \n",
    "    # FIT THE OLS MODELS\n",
    "    LM <- glm(FMLA, data = data,family=binomial) # required for the wild bootstrap\n",
    "    \n",
    "    # SUMMARY OF THE OLS MODEL\n",
    "    cat(\"\\n===============================================\\n\")\n",
    "    cat(\"Model \",model,\":\")\n",
    "    cat(\"\\n===============================================\\n\")\n",
    "   \n",
    "    cat(\"\\n\\nCoefficients from Linear model (adjust_for effects omitted):\\n\")\n",
    "    \n",
    "    # catch the re-ordering of interaction terms introduced by the update(form,dev.new ~ .) call in cluster.wild.glm\n",
    "    tryCatch({\n",
    "        print(coef(summary(LM))[c(\"(Intercept)\",predictors),])    \n",
    "    },error = function(err){\n",
    "        model_coef = dimnames(coef(summary(LM)))[[1]]\n",
    "        cat(\"\\n=------------------------======================\\n\")\n",
    "            print(FMLA)\n",
    "            print(coef(summary(LM)))\n",
    "        cat(\"\\n=------------------------======================\\n\")\n",
    "        cat(\"Missing coefficients:\\n\",\n",
    "            predictors[! predictors %in% model_coef])\n",
    "        cat(\"\\n\\nAvailable coefficients:\\n\")\n",
    "        \n",
    "        print(model_coef)\n",
    "        stop(err)\n",
    "    })\n",
    "\n",
    "    # MAKE THE PRETTY OUTPUT\n",
    "    # without doing the (slow) wild bootstrap\n",
    "    out <- cbind(model=model,\n",
    "                rbind(coef(summary(LM))[predictors,]),\n",
    "                N=data[,.N],\n",
    "                granularity=granularity)\n",
    "\n",
    "    dimnames(out)[[1]] <- predictors\n",
    "    return(list( model=LM, coef=out))\n",
    "\n",
    "} "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the Models for table 2\n",
    "* 2015 data only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===============================================\n",
      "Model  1 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                    Estimate Std. Error   z value      Pr(>|z|)\n",
      "(Intercept)       0.61771643 0.02419287 25.532992 8.483553e-144\n",
      "imputed_new_buses 0.01381816 0.01224066  1.128874  2.589508e-01\n",
      "\n",
      "===============================================\n",
      "Model  2 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                                            Estimate Std. Error   z value\n",
      "(Intercept)                               0.61031688 0.02497743 24.434739\n",
      "imputed_new_buses                         0.04752041 0.01905622  2.493696\n",
      "imputed_new_buses:IsRouteSeattle_10_ride -0.04348364 0.02486266 -1.748954\n",
      "                                              Pr(>|z|)\n",
      "(Intercept)                              7.312118e-132\n",
      "imputed_new_buses                         1.264208e-02\n",
      "imputed_new_buses:IsRouteSeattle_10_ride  8.029901e-02\n",
      "\n",
      "===============================================\n",
      "Model  3 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                                            Estimate Std. Error   z value\n",
      "(Intercept)                               0.81882361 0.03321372 24.653171\n",
      "imputed_new_buses                         0.04342888 0.02082165  2.085756\n",
      "imputed_new_buses:IsRouteSeattle_10_ride -0.04589505 0.02914972 -1.574459\n",
      "                                              Pr(>|z|)\n",
      "(Intercept)                              3.403006e-134\n",
      "imputed_new_buses                         3.700076e-02\n",
      "imputed_new_buses:IsRouteSeattle_10_ride  1.153813e-01\n",
      "\n",
      "===============================================\n",
      "Model  4 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                                            Estimate Std. Error   z value\n",
      "(Intercept)                               0.64018150 0.02397982 26.696672\n",
      "imputed_new_buses                         0.04758349 0.01768485  2.690635\n",
      "imputed_new_buses:IsRouteSeattle_10_ride -0.06334442 0.02424712 -2.612451\n",
      "                                              Pr(>|z|)\n",
      "(Intercept)                              5.144797e-157\n",
      "imputed_new_buses                         7.131613e-03\n",
      "imputed_new_buses:IsRouteSeattle_10_ride  8.989563e-03\n",
      "\n",
      "===============================================\n",
      "Model  5 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                      Estimate Std. Error    z value      Pr(>|z|)\n",
      "(Intercept)        0.585301993 0.01995303 29.3339958 3.823749e-189\n",
      "imputed_new_buses -0.009484197 0.01265283 -0.7495711  4.535131e-01\n",
      "\n",
      "===============================================\n",
      "Model  6 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                     Estimate  Std. Error  z value     Pr(>|z|)\n",
      "(Intercept)       17.14258814 10.48382943 1.635146 0.1020185174\n",
      "imputed_new_buses  0.05212528  0.01547902 3.367479 0.0007585879\n"
     ]
    }
   ],
   "source": [
    "TABLE_2_MODELS_2015 <- list(\n",
    "    \n",
    "        M1 = fit_ridership_logit_models(data = commute_ride_data[year(date) == 2015],\n",
    "                                    predictors=c(\"imputed_new_buses\"),\n",
    "                                    adjust_for=c(route_field,\"f_commutes_since_last_ride\"),\n",
    "                                    model=1,\n",
    "                                    granularity=\"rider/date/commute [2015 only]\"),\n",
    "    \n",
    "        M2 = fit_ridership_logit_models(data = commute_ride_data[year(date) == 2015],\n",
    "                                    predictors=c(\"imputed_new_buses\",\n",
    "                                                 \"imputed_new_buses:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(route_field,\"f_commutes_since_last_ride\"),\n",
    "                                    model=2,\n",
    "                                    granularity=\"rider/date/commute [2015 only]\"),\n",
    "    \n",
    "        M3 = fit_ridership_logit_models(data = commute_ride_data[year(date) == 2015 &  \n",
    "                                                                 first_stop_10_ride >= 0.5],\n",
    "                                    predictors=c(\"imputed_new_buses\",\n",
    "                                                 \"imputed_new_buses:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(route_field,\"f_commutes_since_last_ride\"),\n",
    "                                    model=3,\n",
    "                                    granularity=\"rider/date/commute [2015 only First Stop Preferred Riders]\"),\n",
    "    \n",
    "        M4 = fit_ridership_logit_models(data = commute_ride_data[year(date) == 2015 & \n",
    "                                                                 last_stop_10_ride >= 0.5],\n",
    "                                    predictors=c(\"imputed_new_buses\",\n",
    "                                                 \"imputed_new_buses:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(route_field,\"f_commutes_since_last_ride\"),\n",
    "                                    model=4,\n",
    "                                    granularity=\"rider/date/commute [2015 only; Last AM stop Preferred Riders]\"),\n",
    "    \n",
    "        M5 = fit_ridership_logit_models(data = commute_ride_data[year(date) == 2015 & \n",
    "                                                                 IsRouteSeattle_10_ride >= 0.5],\n",
    "                                    predictors=c(\"imputed_new_buses\"),\n",
    "                                    adjust_for=c(route_field,\"f_commutes_since_last_ride\"),\n",
    "                                    model=5,\n",
    "                                    granularity=\"rider/date/commute [2015 only Seattle Preferred Riders]\"),\n",
    "    \n",
    "        M6 = fit_ridership_logit_models(data = commute_ride_data[year(date) == 2015 & \n",
    "                                                                 IsRouteSeattle_10_ride < 0.5],\n",
    "                                    predictors=c(\"imputed_new_buses\"),\n",
    "                                    adjust_for=c(route_field,\"f_commutes_since_last_ride\"),\n",
    "                                    model=6,\n",
    "                                    granularity=\"rider/date/commute [2015 only; Non-Seattle Preferred Riders]\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table 2 Summary Statistics \n",
    "* Pre 520 HOV data only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>model</th><th scope=col>Estimate</th><th scope=col>Std. Error</th><th scope=col>z value</th><th scope=col>Pr(&gt;|z|)</th><th scope=col>N</th><th scope=col>granularity</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>1                                                            </td><td>0.0138181631384748                                           </td><td>0.0122406555734765                                           </td><td>1.12887443450467                                             </td><td>0.25895080689876                                             </td><td>500000                                                       </td><td>rider/date/commute [2015 only]                               </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>2                                                            </td><td>0.0475204118410548                                           </td><td>0.0190562170291609                                           </td><td>2.49369598217403                                             </td><td>0.0126420772573393                                           </td><td>500000                                                       </td><td>rider/date/commute [2015 only]                               </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses:IsRouteSeattle_10_ride</th><td>2                                                            </td><td>-0.0434836413832317                                          </td><td>0.0248626590673622                                           </td><td>-1.74895377302236                                            </td><td>0.0802990103797351                                           </td><td>500000                                                       </td><td>rider/date/commute [2015 only]                               </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>3                                                            </td><td>0.0434288791693668                                           </td><td>0.0208216517115128                                           </td><td>2.08575572058743                                             </td><td>0.0370007562484731                                           </td><td>336714                                                       </td><td>rider/date/commute [2015 only First Stop Preferred Riders]   </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses:IsRouteSeattle_10_ride</th><td>3                                                            </td><td>-0.045895046512051                                           </td><td>0.0291497187444446                                           </td><td>-1.57445932547112                                            </td><td>0.115381297846007                                            </td><td>336714                                                       </td><td>rider/date/commute [2015 only First Stop Preferred Riders]   </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>4                                                            </td><td>0.0475834857597656                                           </td><td>0.0176848521680221                                           </td><td>2.69063520054787                                             </td><td>0.00713161311559635                                          </td><td>500000                                                       </td><td>rider/date/commute [2015 only; Last AM stop Preferred Riders]</td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses:IsRouteSeattle_10_ride</th><td>4                                                            </td><td>-0.0633444169351657                                          </td><td>0.0242471232648829                                           </td><td>-2.6124508150172                                             </td><td>0.00898956264256854                                          </td><td>500000                                                       </td><td>rider/date/commute [2015 only; Last AM stop Preferred Riders]</td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>5                                                            </td><td>-0.00948419744887215                                         </td><td>0.0126528324890308                                           </td><td>-0.749571090670355                                           </td><td>0.45351306760724                                             </td><td>500000                                                       </td><td>rider/date/commute [2015 only Seattle Preferred Riders]      </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>6                                                            </td><td>0.0521252837562869                                           </td><td>0.0154790224051746                                           </td><td>3.36747905596813                                             </td><td>0.000758587881765966                                         </td><td>284688                                                       </td><td>rider/date/commute [2015 only; Non-Seattle Preferred Riders] </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllll}\n",
       "  & model & Estimate & Std. Error & z value & Pr(>\\textbar{}z\\textbar{}) & N & granularity\\\\\n",
       "\\hline\n",
       "\timputed\\_new\\_buses & 1                                                                 & 0.0138181631384748                                                & 0.0122406555734765                                                & 1.12887443450467                                                  & 0.25895080689876                                                  & 500000                                                            & rider/date/commute {[}2015 only{]}                               \\\\\n",
       "\timputed\\_new\\_buses & 2                                                                 & 0.0475204118410548                                                & 0.0190562170291609                                                & 2.49369598217403                                                  & 0.0126420772573393                                                & 500000                                                            & rider/date/commute {[}2015 only{]}                               \\\\\n",
       "\timputed\\_new\\_buses:IsRouteSeattle\\_10\\_ride & 2                                                                 & -0.0434836413832317                                               & 0.0248626590673622                                                & -1.74895377302236                                                 & 0.0802990103797351                                                & 500000                                                            & rider/date/commute {[}2015 only{]}                               \\\\\n",
       "\timputed\\_new\\_buses & 3                                                                 & 0.0434288791693668                                                & 0.0208216517115128                                                & 2.08575572058743                                                  & 0.0370007562484731                                                & 336714                                                            & rider/date/commute {[}2015 only First Stop Preferred Riders{]}   \\\\\n",
       "\timputed\\_new\\_buses:IsRouteSeattle\\_10\\_ride & 3                                                                 & -0.045895046512051                                                & 0.0291497187444446                                                & -1.57445932547112                                                 & 0.115381297846007                                                 & 336714                                                            & rider/date/commute {[}2015 only First Stop Preferred Riders{]}   \\\\\n",
       "\timputed\\_new\\_buses & 4                                                                 & 0.0475834857597656                                                & 0.0176848521680221                                                & 2.69063520054787                                                  & 0.00713161311559635                                               & 500000                                                            & rider/date/commute {[}2015 only; Last AM stop Preferred Riders{]}\\\\\n",
       "\timputed\\_new\\_buses:IsRouteSeattle\\_10\\_ride & 4                                                                 & -0.0633444169351657                                               & 0.0242471232648829                                                & -2.6124508150172                                                  & 0.00898956264256854                                               & 500000                                                            & rider/date/commute {[}2015 only; Last AM stop Preferred Riders{]}\\\\\n",
       "\timputed\\_new\\_buses & 5                                                                 & -0.00948419744887215                                              & 0.0126528324890308                                                & -0.749571090670355                                                & 0.45351306760724                                                  & 500000                                                            & rider/date/commute {[}2015 only Seattle Preferred Riders{]}      \\\\\n",
       "\timputed\\_new\\_buses & 6                                                                 & 0.0521252837562869                                                & 0.0154790224051746                                                & 3.36747905596813                                                  & 0.000758587881765966                                              & 284688                                                            & rider/date/commute {[}2015 only; Non-Seattle Preferred Riders{]} \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| <!--/--> | model | Estimate | Std. Error | z value | Pr(>|z|) | N | granularity |\n",
       "|---|---|---|---|---|---|---|---|\n",
       "| imputed_new_buses | 1                                                             | 0.0138181631384748                                            | 0.0122406555734765                                            | 1.12887443450467                                              | 0.25895080689876                                              | 500000                                                        | rider/date/commute [2015 only]                                |\n",
       "| imputed_new_buses | 2                                                             | 0.0475204118410548                                            | 0.0190562170291609                                            | 2.49369598217403                                              | 0.0126420772573393                                            | 500000                                                        | rider/date/commute [2015 only]                                |\n",
       "| imputed_new_buses:IsRouteSeattle_10_ride | 2                                                             | -0.0434836413832317                                           | 0.0248626590673622                                            | -1.74895377302236                                             | 0.0802990103797351                                            | 500000                                                        | rider/date/commute [2015 only]                                |\n",
       "| imputed_new_buses | 3                                                             | 0.0434288791693668                                            | 0.0208216517115128                                            | 2.08575572058743                                              | 0.0370007562484731                                            | 336714                                                        | rider/date/commute [2015 only First Stop Preferred Riders]    |\n",
       "| imputed_new_buses:IsRouteSeattle_10_ride | 3                                                             | -0.045895046512051                                            | 0.0291497187444446                                            | -1.57445932547112                                             | 0.115381297846007                                             | 336714                                                        | rider/date/commute [2015 only First Stop Preferred Riders]    |\n",
       "| imputed_new_buses | 4                                                             | 0.0475834857597656                                            | 0.0176848521680221                                            | 2.69063520054787                                              | 0.00713161311559635                                           | 500000                                                        | rider/date/commute [2015 only; Last AM stop Preferred Riders] |\n",
       "| imputed_new_buses:IsRouteSeattle_10_ride | 4                                                             | -0.0633444169351657                                           | 0.0242471232648829                                            | -2.6124508150172                                              | 0.00898956264256854                                           | 500000                                                        | rider/date/commute [2015 only; Last AM stop Preferred Riders] |\n",
       "| imputed_new_buses | 5                                                             | -0.00948419744887215                                          | 0.0126528324890308                                            | -0.749571090670355                                            | 0.45351306760724                                              | 500000                                                        | rider/date/commute [2015 only Seattle Preferred Riders]       |\n",
       "| imputed_new_buses | 6                                                             | 0.0521252837562869                                            | 0.0154790224051746                                            | 3.36747905596813                                              | 0.000758587881765966                                          | 284688                                                        | rider/date/commute [2015 only; Non-Seattle Preferred Riders]  |\n",
       "\n"
      ],
      "text/plain": [
       "                                         model Estimate            \n",
       "imputed_new_buses                        1     0.0138181631384748  \n",
       "imputed_new_buses                        2     0.0475204118410548  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 2     -0.0434836413832317 \n",
       "imputed_new_buses                        3     0.0434288791693668  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 3     -0.045895046512051  \n",
       "imputed_new_buses                        4     0.0475834857597656  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 4     -0.0633444169351657 \n",
       "imputed_new_buses                        5     -0.00948419744887215\n",
       "imputed_new_buses                        6     0.0521252837562869  \n",
       "                                         Std. Error         z value           \n",
       "imputed_new_buses                        0.0122406555734765 1.12887443450467  \n",
       "imputed_new_buses                        0.0190562170291609 2.49369598217403  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.0248626590673622 -1.74895377302236 \n",
       "imputed_new_buses                        0.0208216517115128 2.08575572058743  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.0291497187444446 -1.57445932547112 \n",
       "imputed_new_buses                        0.0176848521680221 2.69063520054787  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.0242471232648829 -2.6124508150172  \n",
       "imputed_new_buses                        0.0126528324890308 -0.749571090670355\n",
       "imputed_new_buses                        0.0154790224051746 3.36747905596813  \n",
       "                                         Pr(>|z|)             N     \n",
       "imputed_new_buses                        0.25895080689876     500000\n",
       "imputed_new_buses                        0.0126420772573393   500000\n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.0802990103797351   500000\n",
       "imputed_new_buses                        0.0370007562484731   336714\n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.115381297846007    336714\n",
       "imputed_new_buses                        0.00713161311559635  500000\n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.00898956264256854  500000\n",
       "imputed_new_buses                        0.45351306760724     500000\n",
       "imputed_new_buses                        0.000758587881765966 284688\n",
       "                                         granularity                                                  \n",
       "imputed_new_buses                        rider/date/commute [2015 only]                               \n",
       "imputed_new_buses                        rider/date/commute [2015 only]                               \n",
       "imputed_new_buses:IsRouteSeattle_10_ride rider/date/commute [2015 only]                               \n",
       "imputed_new_buses                        rider/date/commute [2015 only First Stop Preferred Riders]   \n",
       "imputed_new_buses:IsRouteSeattle_10_ride rider/date/commute [2015 only First Stop Preferred Riders]   \n",
       "imputed_new_buses                        rider/date/commute [2015 only; Last AM stop Preferred Riders]\n",
       "imputed_new_buses:IsRouteSeattle_10_ride rider/date/commute [2015 only; Last AM stop Preferred Riders]\n",
       "imputed_new_buses                        rider/date/commute [2015 only Seattle Preferred Riders]      \n",
       "imputed_new_buses                        rider/date/commute [2015 only; Non-Seattle Preferred Riders] "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "(TABLE_2_2015 <- do.call(rbind,lapply(TABLE_2_MODELS_2015,function(x)x$coef)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the Models for table 2\n",
    "* Pre 520 HOV data only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===============================================\n",
      "Model  1 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                   Estimate  Std. Error  z value     Pr(>|z|)\n",
      "(Intercept)       0.6929681 0.014386498 48.16795 0.000000e+00\n",
      "imputed_new_buses 0.1075117 0.007420369 14.48872 1.427693e-47\n",
      "\n",
      "===============================================\n",
      "Model  2 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                                           Estimate Std. Error   z value\n",
      "(Intercept)                              0.67749614 0.01538975 44.022547\n",
      "imputed_new_buses                        0.08474774 0.01094262  7.744739\n",
      "imputed_new_buses:IsRouteSeattle_10_ride 0.04211266 0.01488042  2.830071\n",
      "                                             Pr(>|z|)\n",
      "(Intercept)                              0.000000e+00\n",
      "imputed_new_buses                        9.577850e-15\n",
      "imputed_new_buses:IsRouteSeattle_10_ride 4.653769e-03\n",
      "\n",
      "===============================================\n",
      "Model  3 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                                            Estimate Std. Error    z value\n",
      "(Intercept)                              0.814905626 0.02506317 32.5140743\n",
      "imputed_new_buses                        0.130726229 0.01494760  8.7456332\n",
      "imputed_new_buses:IsRouteSeattle_10_ride 0.007184637 0.02178801  0.3297518\n",
      "                                              Pr(>|z|)\n",
      "(Intercept)                              6.745123e-232\n",
      "imputed_new_buses                         2.217685e-18\n",
      "imputed_new_buses:IsRouteSeattle_10_ride  7.415875e-01\n",
      "\n",
      "===============================================\n",
      "Model  4 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                                            Estimate Std. Error    z value\n",
      "(Intercept)                              0.814905626 0.02506317 32.5140743\n",
      "imputed_new_buses                        0.130726229 0.01494760  8.7456332\n",
      "imputed_new_buses:IsRouteSeattle_10_ride 0.007184637 0.02178801  0.3297518\n",
      "                                              Pr(>|z|)\n",
      "(Intercept)                              6.745123e-232\n",
      "imputed_new_buses                         2.217685e-18\n",
      "imputed_new_buses:IsRouteSeattle_10_ride  7.415875e-01\n",
      "\n",
      "===============================================\n",
      "Model  5 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                   Estimate Std. Error  z value     Pr(>|z|)\n",
      "(Intercept)       0.6316045 0.01531256 41.24748 0.000000e+00\n",
      "imputed_new_buses 0.1264590 0.01000736 12.63659 1.326808e-36\n",
      "\n",
      "===============================================\n",
      "Model  6 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                    Estimate Std. Error    z value     Pr(>|z|)\n",
      "(Intercept)       0.26682460 2.73942893 0.09740154 9.224075e-01\n",
      "imputed_new_buses 0.08938789 0.01115749 8.01147014 1.133453e-15\n"
     ]
    }
   ],
   "source": [
    "TABLE_2_MODELS_pre_hov <- list(\n",
    "    \n",
    "        M1 = fit_ridership_logit_models(data = commute_ride_data[(!SR520WBHOV) & \n",
    "                                                                 (!SR520EBHOV)],\n",
    "                                    predictors=c(\"imputed_new_buses\"),\n",
    "                                    adjust_for=c(route_field,\n",
    "                                                 \"f_commutes_since_last_ride\"),\n",
    "                                    model=1,\n",
    "                                    granularity=\"rider/date/commute [Pre-HOV]\"),\n",
    "    \n",
    "        M2 = fit_ridership_logit_models(data = commute_ride_data[(!SR520WBHOV) & \n",
    "                                                                 (!SR520EBHOV)],\n",
    "                                    predictors=c(\"imputed_new_buses\",\n",
    "                                                 \"imputed_new_buses:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(route_field,\n",
    "                                                 \"f_commutes_since_last_ride\"),\n",
    "                                    model=2,\n",
    "                                    granularity=\"rider/date/commute [Pre-HOV]\"),\n",
    "   \n",
    "        M3 = fit_ridership_logit_models(data = commute_ride_data[(!SR520WBHOV) &  \n",
    "                                                                 (!SR520EBHOV) & \n",
    "                                                                 (first_stop_10_ride >= 0.5)],\n",
    "                                    predictors=c(\"imputed_new_buses\",\n",
    "                                                 \"imputed_new_buses:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(route_field,\n",
    "                                                 \"f_commutes_since_last_ride\"),\n",
    "                                    model=3,\n",
    "                                    granularity=\"rider/date/commute [Pre-HOV; First stop preferred riders]\"),\n",
    "    \n",
    "        M4 = fit_ridership_logit_models(data = commute_ride_data[(!SR520WBHOV) & \n",
    "                                                                 (!SR520EBHOV) & \n",
    "                                                                 (first_stop_10_ride >= 0.5)],\n",
    "                                    predictors=c(\"imputed_new_buses\",\n",
    "                                                 \"imputed_new_buses:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(route_field,\n",
    "                                                 \"f_commutes_since_last_ride\"),\n",
    "                                    model=4,\n",
    "                                    granularity=\"rider/date/commute [Pre-HOV; Last stop preferred riders]\"),\n",
    "    \n",
    "        M5 = fit_ridership_logit_models(data = commute_ride_data[(!SR520WBHOV) & \n",
    "                                                                 (!SR520EBHOV) & \n",
    "                                                                 IsRouteSeattle_10_ride >= 0.5],\n",
    "                                    predictors=c(\"imputed_new_buses\"),\n",
    "                                    adjust_for=c(route_field,\n",
    "                                                 \"f_commutes_since_last_ride\"),\n",
    "                                    model=5,\n",
    "                                    granularity=\"rider/date/commute [Pre-HOV; Seattle preferred riders]\"),\n",
    "    \n",
    "        M6 = fit_ridership_logit_models(data = commute_ride_data[(!SR520WBHOV) & \n",
    "                                                                 (!SR520EBHOV) & \n",
    "                                                                 IsRouteSeattle_10_ride <= 0.5],\n",
    "                                    predictors=c(\"imputed_new_buses\"),\n",
    "                                    adjust_for=c(route_field,\n",
    "                                                 \"f_commutes_since_last_ride\"),\n",
    "                                    model=6,\n",
    "                                    granularity=\"rider/date/commute [Pre-HOV; Non-Seattle preferred riders]\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table 2 Summary Statistics \n",
    "* Pre 520 HOV data only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>model</th><th scope=col>Estimate</th><th scope=col>Std. Error</th><th scope=col>z value</th><th scope=col>Pr(&gt;|z|)</th><th scope=col>N</th><th scope=col>granularity</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>1                                                         </td><td>0.107511680330711                                         </td><td>0.00742036886571824                                       </td><td>14.4887245198025                                          </td><td>1.42769317588814e-47                                      </td><td>1793450                                                   </td><td>rider/date/commute [Pre-HOV]                              </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>2                                                         </td><td>0.0847477439068121                                        </td><td>0.0109426212644368                                        </td><td>7.74473883896906                                          </td><td>9.57785014827185e-15                                      </td><td>1793450                                                   </td><td>rider/date/commute [Pre-HOV]                              </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses:IsRouteSeattle_10_ride</th><td>2                                                         </td><td>0.0421126552447275                                        </td><td>0.0148804239797333                                        </td><td>2.83007092419436                                          </td><td>0.00465376864951135                                       </td><td>1793450                                                   </td><td>rider/date/commute [Pre-HOV]                              </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>3                                                         </td><td>0.130726229237663                                         </td><td>0.0149476003256684                                        </td><td>8.74563317117709                                          </td><td>2.21768496275528e-18                                      </td><td>778988                                                    </td><td>rider/date/commute [Pre-HOV; First stop preferred riders] </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses:IsRouteSeattle_10_ride</th><td>3                                                         </td><td>0.00718463716979866                                       </td><td>0.0217880137151369                                        </td><td>0.329751819680894                                         </td><td>0.741587495181016                                         </td><td>778988                                                    </td><td>rider/date/commute [Pre-HOV; First stop preferred riders] </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>4                                                         </td><td>0.130726229237663                                         </td><td>0.0149476003256684                                        </td><td>8.74563317117709                                          </td><td>2.21768496275528e-18                                      </td><td>778988                                                    </td><td>rider/date/commute [Pre-HOV; Last stop preferred riders]  </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses:IsRouteSeattle_10_ride</th><td>4                                                         </td><td>0.00718463716979866                                       </td><td>0.0217880137151369                                        </td><td>0.329751819680894                                         </td><td>0.741587495181016                                         </td><td>778988                                                    </td><td>rider/date/commute [Pre-HOV; Last stop preferred riders]  </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>5                                                         </td><td>0.126458955824258                                         </td><td>0.0100073621929494                                        </td><td>12.6365922793674                                          </td><td>1.32680762823826e-36                                      </td><td>1129688                                                   </td><td>rider/date/commute [Pre-HOV; Seattle preferred riders]    </td></tr>\n",
       "\t<tr><th scope=row>imputed_new_buses</th><td>6                                                         </td><td>0.0893878933094686                                        </td><td>0.0111574894176531                                        </td><td>8.01147014023074                                          </td><td>1.13345256731885e-15                                      </td><td>665328                                                    </td><td>rider/date/commute [Pre-HOV; Non-Seattle preferred riders]</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllll}\n",
       "  & model & Estimate & Std. Error & z value & Pr(>\\textbar{}z\\textbar{}) & N & granularity\\\\\n",
       "\\hline\n",
       "\timputed\\_new\\_buses & 1                                                              & 0.107511680330711                                              & 0.00742036886571824                                            & 14.4887245198025                                               & 1.42769317588814e-47                                           & 1793450                                                        & rider/date/commute {[}Pre-HOV{]}                              \\\\\n",
       "\timputed\\_new\\_buses & 2                                                              & 0.0847477439068121                                             & 0.0109426212644368                                             & 7.74473883896906                                               & 9.57785014827185e-15                                           & 1793450                                                        & rider/date/commute {[}Pre-HOV{]}                              \\\\\n",
       "\timputed\\_new\\_buses:IsRouteSeattle\\_10\\_ride & 2                                                              & 0.0421126552447275                                             & 0.0148804239797333                                             & 2.83007092419436                                               & 0.00465376864951135                                            & 1793450                                                        & rider/date/commute {[}Pre-HOV{]}                              \\\\\n",
       "\timputed\\_new\\_buses & 3                                                              & 0.130726229237663                                              & 0.0149476003256684                                             & 8.74563317117709                                               & 2.21768496275528e-18                                           & 778988                                                         & rider/date/commute {[}Pre-HOV; First stop preferred riders{]} \\\\\n",
       "\timputed\\_new\\_buses:IsRouteSeattle\\_10\\_ride & 3                                                              & 0.00718463716979866                                            & 0.0217880137151369                                             & 0.329751819680894                                              & 0.741587495181016                                              & 778988                                                         & rider/date/commute {[}Pre-HOV; First stop preferred riders{]} \\\\\n",
       "\timputed\\_new\\_buses & 4                                                              & 0.130726229237663                                              & 0.0149476003256684                                             & 8.74563317117709                                               & 2.21768496275528e-18                                           & 778988                                                         & rider/date/commute {[}Pre-HOV; Last stop preferred riders{]}  \\\\\n",
       "\timputed\\_new\\_buses:IsRouteSeattle\\_10\\_ride & 4                                                              & 0.00718463716979866                                            & 0.0217880137151369                                             & 0.329751819680894                                              & 0.741587495181016                                              & 778988                                                         & rider/date/commute {[}Pre-HOV; Last stop preferred riders{]}  \\\\\n",
       "\timputed\\_new\\_buses & 5                                                              & 0.126458955824258                                              & 0.0100073621929494                                             & 12.6365922793674                                               & 1.32680762823826e-36                                           & 1129688                                                        & rider/date/commute {[}Pre-HOV; Seattle preferred riders{]}    \\\\\n",
       "\timputed\\_new\\_buses & 6                                                              & 0.0893878933094686                                             & 0.0111574894176531                                             & 8.01147014023074                                               & 1.13345256731885e-15                                           & 665328                                                         & rider/date/commute {[}Pre-HOV; Non-Seattle preferred riders{]}\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| <!--/--> | model | Estimate | Std. Error | z value | Pr(>|z|) | N | granularity |\n",
       "|---|---|---|---|---|---|---|---|\n",
       "| imputed_new_buses | 1                                                          | 0.107511680330711                                          | 0.00742036886571824                                        | 14.4887245198025                                           | 1.42769317588814e-47                                       | 1793450                                                    | rider/date/commute [Pre-HOV]                               |\n",
       "| imputed_new_buses | 2                                                          | 0.0847477439068121                                         | 0.0109426212644368                                         | 7.74473883896906                                           | 9.57785014827185e-15                                       | 1793450                                                    | rider/date/commute [Pre-HOV]                               |\n",
       "| imputed_new_buses:IsRouteSeattle_10_ride | 2                                                          | 0.0421126552447275                                         | 0.0148804239797333                                         | 2.83007092419436                                           | 0.00465376864951135                                        | 1793450                                                    | rider/date/commute [Pre-HOV]                               |\n",
       "| imputed_new_buses | 3                                                          | 0.130726229237663                                          | 0.0149476003256684                                         | 8.74563317117709                                           | 2.21768496275528e-18                                       | 778988                                                     | rider/date/commute [Pre-HOV; First stop preferred riders]  |\n",
       "| imputed_new_buses:IsRouteSeattle_10_ride | 3                                                          | 0.00718463716979866                                        | 0.0217880137151369                                         | 0.329751819680894                                          | 0.741587495181016                                          | 778988                                                     | rider/date/commute [Pre-HOV; First stop preferred riders]  |\n",
       "| imputed_new_buses | 4                                                          | 0.130726229237663                                          | 0.0149476003256684                                         | 8.74563317117709                                           | 2.21768496275528e-18                                       | 778988                                                     | rider/date/commute [Pre-HOV; Last stop preferred riders]   |\n",
       "| imputed_new_buses:IsRouteSeattle_10_ride | 4                                                          | 0.00718463716979866                                        | 0.0217880137151369                                         | 0.329751819680894                                          | 0.741587495181016                                          | 778988                                                     | rider/date/commute [Pre-HOV; Last stop preferred riders]   |\n",
       "| imputed_new_buses | 5                                                          | 0.126458955824258                                          | 0.0100073621929494                                         | 12.6365922793674                                           | 1.32680762823826e-36                                       | 1129688                                                    | rider/date/commute [Pre-HOV; Seattle preferred riders]     |\n",
       "| imputed_new_buses | 6                                                          | 0.0893878933094686                                         | 0.0111574894176531                                         | 8.01147014023074                                           | 1.13345256731885e-15                                       | 665328                                                     | rider/date/commute [Pre-HOV; Non-Seattle preferred riders] |\n",
       "\n"
      ],
      "text/plain": [
       "                                         model Estimate           \n",
       "imputed_new_buses                        1     0.107511680330711  \n",
       "imputed_new_buses                        2     0.0847477439068121 \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 2     0.0421126552447275 \n",
       "imputed_new_buses                        3     0.130726229237663  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 3     0.00718463716979866\n",
       "imputed_new_buses                        4     0.130726229237663  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 4     0.00718463716979866\n",
       "imputed_new_buses                        5     0.126458955824258  \n",
       "imputed_new_buses                        6     0.0893878933094686 \n",
       "                                         Std. Error          z value          \n",
       "imputed_new_buses                        0.00742036886571824 14.4887245198025 \n",
       "imputed_new_buses                        0.0109426212644368  7.74473883896906 \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.0148804239797333  2.83007092419436 \n",
       "imputed_new_buses                        0.0149476003256684  8.74563317117709 \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.0217880137151369  0.329751819680894\n",
       "imputed_new_buses                        0.0149476003256684  8.74563317117709 \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.0217880137151369  0.329751819680894\n",
       "imputed_new_buses                        0.0100073621929494  12.6365922793674 \n",
       "imputed_new_buses                        0.0111574894176531  8.01147014023074 \n",
       "                                         Pr(>|z|)             N      \n",
       "imputed_new_buses                        1.42769317588814e-47 1793450\n",
       "imputed_new_buses                        9.57785014827185e-15 1793450\n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.00465376864951135  1793450\n",
       "imputed_new_buses                        2.21768496275528e-18 778988 \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.741587495181016    778988 \n",
       "imputed_new_buses                        2.21768496275528e-18 778988 \n",
       "imputed_new_buses:IsRouteSeattle_10_ride 0.741587495181016    778988 \n",
       "imputed_new_buses                        1.32680762823826e-36 1129688\n",
       "imputed_new_buses                        1.13345256731885e-15 665328 \n",
       "                                         granularity                                               \n",
       "imputed_new_buses                        rider/date/commute [Pre-HOV]                              \n",
       "imputed_new_buses                        rider/date/commute [Pre-HOV]                              \n",
       "imputed_new_buses:IsRouteSeattle_10_ride rider/date/commute [Pre-HOV]                              \n",
       "imputed_new_buses                        rider/date/commute [Pre-HOV; First stop preferred riders] \n",
       "imputed_new_buses:IsRouteSeattle_10_ride rider/date/commute [Pre-HOV; First stop preferred riders] \n",
       "imputed_new_buses                        rider/date/commute [Pre-HOV; Last stop preferred riders]  \n",
       "imputed_new_buses:IsRouteSeattle_10_ride rider/date/commute [Pre-HOV; Last stop preferred riders]  \n",
       "imputed_new_buses                        rider/date/commute [Pre-HOV; Seattle preferred riders]    \n",
       "imputed_new_buses                        rider/date/commute [Pre-HOV; Non-Seattle preferred riders]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "(TABLE_2_pre_hov <- do.call(rbind,lapply(TABLE_2_MODELS_pre_hov,function(x)x$coef)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the Models for Table 3 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===============================================\n",
      "Model  1 :\n",
      "===============================================\n",
      "\n",
      "\n",
      "Coefficients from Linear model (adjust_for effects omitted):\n",
      "                                 Estimate  Std. Error  z value    Pr(>|z|)\n",
      "(Intercept)                     0.6416053 0.032314269 19.85517 9.94239e-88\n",
      "post_hov:IsRouteSeattle_10_ride 0.1066583 0.007100197 15.02188 5.27879e-51\n"
     ]
    }
   ],
   "source": [
    "commute_ride_data[,post_hov :=  0 + ((AfternoonId ==0 & SR520WBHOV) | (AfternoonId ==1 & SR520EBHOV))]\n",
    "\n",
    "TABLE_3_MODELS <- list(\n",
    "    \n",
    "        M1 = fit_ridership_logit_models(data = commute_ride_data,\n",
    "                                    predictors=c(\"post_hov:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(\"imputed_new_buses + AfternoonId + fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=1,\n",
    "                                    granularity=\"rider/date/commute\"),\n",
    "    \n",
    "        M2 = fit_ridership_logit_models(data = commute_ride_data,\n",
    "                                    predictors=c(\"post_hov:IsRouteSeattle_10_ride\",\n",
    "                                                 \"is_peak_10_ride\",\n",
    "                                                 \"IsRouteSeattle_10_ride:is_peak_10_ride\",\n",
    "                                                 \"post_hov:is_peak_10_ride\",\n",
    "                                                 \"post_hov:IsRouteSeattle_10_ride:is_peak_10_ride\"\n",
    "                                            ),\n",
    "                                    model=2,\n",
    "                                    adjust_for=c(\"imputed_new_buses + AfternoonId + fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    granularity=\"rider/date/commute\"),\n",
    "    \n",
    "        M3 = fit_ridership_logit_models(data = commute_ride_data[first_stop_10_ride >= 0.5],\n",
    "                                    predictors=c(\"post_hov:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(\"imputed_new_buses\",\n",
    "                                                 \"fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=3,\n",
    "                                    granularity=\"rider/date/commute [first stop prefered riders]\"),\n",
    "    \n",
    "        M4 = fit_ridership_logit_models(data = commute_ride_data[last_stop_10_ride >= 0.5],\n",
    "                                    predictors=c(\"post_hov:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(\"imputed_new_buses\",\n",
    "                                                 \"fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=4,\n",
    "                                    granularity=\"rider/date/commute [last stop prefered riders]\"),\n",
    "    \n",
    "        M5 = fit_ridership_logit_models(data = commute_ride_data[date >= as.Date(\"2016-03-01\") & \n",
    "                                                                  date < as.Date(\"2016-07-01\")],\n",
    "                                    predictors=c(\"post_hov:IsRouteSeattle_10_ride\"),\n",
    "                                    adjust_for=c(\"imputed_new_buses\",\n",
    "                                                 \"fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=5,\n",
    "                                    granularity=\"ride/date/commute\")\n",
    "                                    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table 3 Summary Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(TABLE_3 <- do.call(rbind,lapply(TABLE_3_MODELS,function(x)x$coef)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the Models for Table 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TABLE_4_MODELS <- list(\n",
    "    \n",
    "        M1 = fit_ridership_logit_models(data = commute_ride_data[(single_stop_10_ride >= 0.5)],\n",
    "                                    predictors=c(\"imputed_new_buses\"),\n",
    "                                    adjust_for=c(\"fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=1,\n",
    "                                    granularity=\"rider/date/commute [Single Stop Preferred Riders]\"),\n",
    "    \n",
    "        M2 = fit_ridership_logit_models(data = commute_ride_data,\n",
    "                                    predictors=c(\"imputed_new_buses\",\n",
    "                                                 \"imputed_new_buses:single_stop_10_ride\"\n",
    "                                                ),\n",
    "                                    adjust_for=c(\"imputed_new_buses + fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=2,\n",
    "                                    granularity=\"rider/date/commute\"),\n",
    "    \n",
    "        M3 = fit_ridership_logit_models(data = commute_ride_data,\n",
    "                                    predictors=c(\n",
    "                                                \"imputed_new_buses\",\n",
    "                                                \"Is405North_10_ride:I405HOV\",\n",
    "                                                \"imputed_new_buses:I405HOV\",\n",
    "                                                \"imputed_new_buses:Is405North_10_ride\",\n",
    "                                                \"imputed_new_buses:Is405North_10_ride:I405HOV\"\n",
    "                                                ),\n",
    "                                    adjust_for=c(\"imputed_new_buses + fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=3,\n",
    "                                    granularity=\"rider/date/commute\"),\n",
    "\n",
    "        M4 = fit_ridership_logit_models(data = commute_ride_data,\n",
    "                                    predictors=c(\n",
    "                                                \"imputed_new_buses\",\n",
    "                                                \"imputed_new_buses:single_stop_10_ride\",\n",
    "                                                \"Is405North_10_ride:I405HOV\",\n",
    "                                                \"imputed_new_buses:Is405North_10_ride:I405HOV\",\n",
    "                                                \"single_stop_10_ride:Is405North_10_ride:I405HOV\"\n",
    "                                                ),\n",
    "                                    adjust_for=c(\"imputed_new_buses + fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=4,\n",
    "                                    granularity=\"rider/date/commute\"),\n",
    "\n",
    "        M5 = fit_ridership_logit_models(data = commute_ride_data,\n",
    "                                    predictors=c(\n",
    "                                                \"imputed_new_buses\",\n",
    "                                                \"imputed_new_buses:single_stop_10_ride\",\n",
    "                                                \"Is405North_10_ride:I405HOV\",\n",
    "                                                \"imputed_new_buses:Is405North_10_ride:I405HOV\",\n",
    "                                                \"single_stop_10_ride:Is405North_10_ride:I405HOV\",\n",
    "                                                \"imputed_new_buses:single_stop_10_ride:Is405North_10_ride:I405HOV\"\n",
    "                                                ),\n",
    "                                    adjust_for=c(\"imputed_new_buses + fDate\",\n",
    "                                                 \"f_commutes_since_last_ride\",\n",
    "                                                 route_field),\n",
    "                                    model=5,\n",
    "                                    granularity=\"rider/date/commute\")\n",
    "                                    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table 4 Summary Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(TABLE_4 <- do.call(rbind,lapply(TABLE_4_MODELS,function(x)x$coef)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "1 + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
